Code-coverage-based test data adequacy criteria typically treat all coverable code elements (such as statements, basic blocks or outcomes of decisions) as equal. In practice, however, the probability that a test case can expose a fault in a code element varies: some faults are more easily revealed than others. Thus, several researchers have suggested that if one could estimate the probability that a fault in a code element will cause a failure, one could use this estimate to determine the number of executions of a code element that are required to achieve a certain level of confidence in that element's correctness. This estimate, in turn, could be used to improve the fault-detection effectiveness of test suites and help testers distribute testing resources more effectively. This conjecture is intriguing; however, like many such conjectures it has never been directly examined empirically. If empirical evidence were to support this conjecture, it would motivate further research into methodologies for obtaining fault-exposure-potential estimates and incorporating them into test data adequacy criteria. This paper reports the results of experiments conducted to investigate the effects of incorporating an estimate of fault-exposure probability into the statement coverage test data adequacy criterion. The results of these experiments, however, ran contrary to the conjectures of previous researchers. Although incorporation of the estimates did produce statistically significant increases in the fault-detection effectiveness of test suites, these increases were quite small, suggesting that the approach might not be able to produce the gains hoped for and might not be worth the cost of its employment.

Available soon...

Available soon...

Available soon...

Mutation testing is a technique for unit testing software that, although powerful, is computationally expensive. The principal expense of mutation is that many variants of the test program, called mutants, must be repeatedly executed. This paper quanti es the expense of mutation in terms of the number of mutants that are created, then proposes and evaluates a technique that reduces the number of mutants by an order of magnitude. Selective mutation reduces the cost of mutation testing by reducing the number of mutants. This paper reports experimental results that compare selective mutation testing with standard, or non-selective, mutation testing, and results that quantify the savings achieved by selective mutation testing. The results support the hypothesis that selective mutation is almost as strong as non-selective mutation; in experimental trials selective mutation provides almost the same coverage as non-selective mutation, with a four-fold or more reduction in the number of mutants.

Mutation-based software testing, or \em mutation testing, is a powerful testing technique applied primarily at the unit software level. Central to mutation testing is the need to analyze a test set to determine a quality measure called the \em mutation adequacy score\,; this assessment process is called \em mutation analysis. Unfortunately, the conventional method of performing mutation analysis, which requires interpreting many slightly different versions of the same program, has significant problems. Automated mutation analysis systems based on the conventional interpretive method are slow, laborious to build, and usually unable to completely emulate the intended operational environment of the software being tested. This research presents a solution to these problems: the \underlineMutant \underlineSchema \underlineGeneration (MSG) method. Rather than mutating an intermediate form of the program that then must be interpreted, this new method describes how to encode all mutations into one source-level program, a ``metamutant''\@. This program is then compiled (once) with the same compiler used during development and is executed in the same operational environment at compiled-program speeds. Since mutation systems based on mutant schemata do not need to provide run-time semantics and environment, they are significantly less complex and easier to build than interpretive systems, as well as more portable. An approach to automatically generating metamutants using attribute grammars is also presented. An MSG-based prototype mutation analysis system, \tt TUMS, was designed and implemented to demonstrate the automated generation of metamutants and to allow empirical performance studies to be conducted. Benchmarks show \tt TUMS significantly faster than \tt Mothra, a conventional interpretive mutation analysis system, with speed-ups as high as an order-of-magnitude observed. Additional studies are reported that contrast the performance of \tt TUMS to a hypothetical ``ideal'' mutation analysis system. We conclude that high performance mutation analysis is possible through the \mboxcreation and instantiation of mutant schemata and that the MSG method described in this dissertation is a viable and desirable approach for building automated mutation analysis systems.

Available soon...

Mutation analysis is a powerful technique for assessing and improving the quality of test data used to unit test software. Unfortunately, current automated mutation analysis systems suffer from severe performance problems. This paper presents a new method for performing mutation analysis that uses program schemata to encode all mutants for a program into one metaprogram, which is subsequently compiled and run at speeds substantially higher than achieved by previous interpretive systems. Preliminary performance improvements of over 300% are reported. This method has the additional advantages of being easier to implement than interpretive systems, being simpler to port across a wide range of hardware and software platforms, and using the same compiler and run-time support system that is used during development and/or deployment.

Available soon...